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-  ,  ABSTRACT 

'  ni  t  , 

-Leonard— H-983-}- proposes  a  design  measure  which  may  be  used  sequentially 

to  choose  the  next  dose  level  in  a  linear  logistic  quantal  response  model  for 

bioassay.  His  design  measure  averages  the  posterior  distribution  of  the 

(~P  i  t  olo-Sw 

effective  dose  over  those  -LD’  values  which  are  regarded  as  important.  In 

this  evaluation  the  mode  of  the  design  density  is  used  as  the  next  design 

to  GP  GO 

point,  and  it  is  supposed  that  all  -LB-  values  between  -itr  60  and  -Hr  90  are 

equally  important.  After  ten  initial  badly  designed  observations,  it  is  shown 

that  only  20  further,  well  designed,  observations  are  needed  to  obtain  a 

design  efficiency  of  about  82%,  and  an  estimated  response  curve  which  lies  at 

GO  <~p 

a  maximum  of  an  estimated  6-&D* points  from  the  true  curve,  for  all  iXT  values 

Go  ex> 

lying  between  -LB-  60  and  -LB  90.  If  more  observations  are  taken  then  the 
design  efficiency  increases  steadily,  but  it  is  difficult  to  increase  the 
accuracy  of  estimation  without  either  taking  many  more  observations,  or  by 
pushing  the  design  points  outside  the  appropriate  region.  However,  within  the 
design  region,  chosen  by  any  recommended  procedure,  the  method  promises 
excellent  robustness,  with  respect  to  possible  inadequacies  in  the  model, 
whilst  outlying  design  points  would  not  provide  such  robustness. 
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SIGNIFICANCE  AND  EXPLANATION 


Leonard  (1982)  proposes  a  design  measure  for  the  quantal  response  model, 
based  upon  a  mixture  of  posterior  distributions  of  the  effective  doses.  In 
this  paper  the  mode  of  this  distribution  is  evaluated  sequentially  according 
to  the  criteria  of  "estimation  accuracy"  and  "design  efficiency"  which  relate 
to  both  good  estimation  conditional  on  the  model,  and  good  robustness  with 
respect  to  deviations  from  the  model.  For  sequential  design  points,  for  ED 
values  between  ED  60  and  ED  90,  it  is  shown  by  simulations  that,  after  an 
initial  batch  of  ten  badly  chosen  design  points,  only  20  further  observations 
are  needed  to  achieve  a  design  efficiency  of  82%  and  an  estimated  response 
curve  which  lies  at  a  maximum  of  an  estimated  6  ED  points  from  the  true 
curve.  For  more  than  30  observations  it  is  necessary  to  spread  out  the  design 
points  more,  in  order  to  sacrifice  some  design  efficiency  for  more  estimation 
accuracy.  Further  simulations  are  currently  being  carried  out  by  Leonard  and 
Hamada,  to  investigate  this  aspect  more  closely. 


Accession  For 

NT I 3  GRA&I 

DTIC  TAB 

Unannounced 

□ 

Justification _ 

By 

Distribution/ 
Availability  Codes 
(Avail  and/or 
Dist  |  Special 


iA 


The  responsibility  for  the  wording  and  views  expressed  in  this  descriptive 
summary  lies  with  MRC,  and  not  with  the  author  of  this  report. 


A  SMALL  SAMPLE  EVALUATION  OF  A  BAYESIAN  DESIGN  METHOD 
FOR  QUANTAL  RESPONSE  MODELS 
Tom  Leonard 


1 .  INTRODUCTION 

Consider  zero-case  responses  y.j,...,yn  which  are  independent  binary 
variables,  with 

p(yi  =  1 | ©i )  =  0i  (i  =  1 , . . . ,n)  .  (1.1) 

Assume  the  linear  logistic  model 

a  =  log  0^  -  log(  1  -  0^)  =  +  g^x^  (1.2) 

where  x^,...,x,  will  be  referred  to  as  "dose  levels". 

We  address  the  following  three  problems: 

(i)  Estimation  of  the  response  curve 


W  B0+eiX 

0(x)  =  e  U  /(I  +  e  ) 


(1.3) 


after  n  observations. 


(ii)  Choice  of  the  next  design  point,  xn+i»  given  ylf...,yn  and 

x^,...,xn,  and  the  sequential  choice  of  xn+r/  given  y-j > • • • 'Yn+r-1 
and  x^,...,xn+J — -j ,  for  r  =  1,2,...  . 

(iii)  The  problem  in  (ii)  but  where  the  results,  for  values  of  0  in  (1.3) 
lying  within  specified  limits  (a,b),  are  robust  against  possible 
inadequacies  of  the  model  in  (1.2). 

These  problems  may  be  tackled  as  follows: 

(i)  The  response  curve  in  (1.3)  will  be  estimated  by  the  approximation 

recommended  by  Leonard  (1982)  to  the  posterior  mean  value  function  of 
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the  curve.  This  mean  value  function  is  vastly  superior  in  terms  of 
accuracy  of  estimation  to  the  maximum  likelihood  estimate  of  the 
response  curve,  and  is  primarily  responsible  for  our  good  practical 
results.  It  is  also  undoubtedly  superior  to  non-parametric  estimates 
based  upon  Dirichlet  processes,  since  the  latter  are  not  smooth  enough 
to  reasonably  approximate  a  smooth  true  response  curve,  and  also  depend 
upon  a  number  of  complicated  prior  parameter  values. 

(ii)  For  illustrative  purposes  it  is  supposed  that  the  function  in  (1.3)  is 
of  interest  for  values  of  0  lying  between  0.6  and  0.9.  At  any  stage 
in  the  experiment,  the  next  design  point  will  be  chosen  to  be  the  mode 
of  the  design  density  recommended  by  Leonard  (1982),  given  all 
observations  up  to  that  point.  This  density  averages  the  posterior 
densities  of  the  effective  closes  over  LD  values  60  to  90.  It  may  be 
adjusted  to  any  0  region  of  interest. 

(iii)  The  design  procedure  in  (ii)  attempts  to  choose  x  values  such  that 
the  corresponding  0(x)  values  have  the  best  chance  of  lying  between 
0.6  and  0.9.  Note  that 

(A)  Conditional  on  the  model  better  estimates  of  this  region  of  the  response 
curve  may  be  obtained  by  taking  x  points  further  apart. 

However, 

(B)  Results  based  upon  widely  spread  x  points  will  not  be  robust  under 
inadequacies  in  the  model.  Therefore  it  is  sensible  to  define  design 
percentage  efficiency  as  the  long  run  percentage  of  x  values  such  that  for 
the  true  curve  0,  the  corresponding  0(x)  values  lie  between  0.6  and  0.9. 

A  high  percentage  design  efficiency  means  good  estimation  and  excellent 
robustness,  whilst  a  low  efficiency  could  mean  excellent  estimation 
conditional  on  the  model,  but  bad  robustness. 


★ 

For  an  estimate  0  of  the  true  response  curve  8,  the  estimation 
accuracy  is  defined  to  be 

DIFFMAX  =  max  |9*(x)  -  0(x)|  .  (1.4) 

0.6<6(x) <0.9 

The  clear  objective  should  be  to  obtain  good  estimation  accuracy  whilst 
at  the  same  time  maintaining  high  percentage  design  efficiency.  There  is  a 
trade-off  between  estimation  accuracy  and  design  efficiency?  the  objective  is 
to  obtain  good  estimates  of  9  which  are  also  robust  against  changes  in  the 
model. 

2.  A  SMALL  SAMPLE  EVALUATION,  AND  OVERALL  CONCLUSIONS 

The  method  described  in  Section  1  was  evaluated  based  upon  a  true 
response  curve  taking  the  linear  logistic  form  in  (1.3)  with  3Q  =  -4.39  and 
=  8.79.  Owing  to  properties  of  the  curve,  under  location  and  scale 
transformations,  the  choices  of  3Q  and  31  can  be  made  arbitrarily.  The 
particular  choices  made  ensure  that  9(0.5)  =  0.5  and  0(0.9)  =  0.75. 

In  each  simulation  the  ten  initial  x's  were  taken  co  be 
0,0.05,0. 1 ,0 . 15, ... ,0.45  corresponding  to  0  values  ranging  between  0.012  and 
Q.392s  These  points  are  badly  designed  by  intention  in  order  to  give  worst- 
case  results.  Our  estimation  accuracies  and  design  efficiencies  could  be 
noticeably  improved  by  basing  the  first  few  choices  on  sensible  reasoning 
relative  to  the  practical  situation  under  consideration. 

The  estimation  accuracies  for  10  runs,  each  with  30  further  observations 
based  upon  well  (sequentially)  designed  x  points,  are  described  in  Table  1. 
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Table  1:  Estimation  Accuracies  for  up  to  30  Further  Observations 


Sample  Size 

10 

15 

20 

25 

30 

Run  1 

0.05 

0.05 

0.04 

0.03 

0.02 

Run  2 

0.15 

0.09 

0.07 

0.02 

0.02 

Run  3 

0.12 

0.08 

0.06 

0.  11 

0.06 

Run  4 

0.11 

0.13 

0.10 

0.09 

0.12 

Run  5 

0.06 

0.04 

0.02 

0.04 

0.09 

Run  6 

0.03 

0.01 

0.03 

0.03 

0.02 

Run  7 

0.15 

0.06 

0.04 

0.03 

0.06 

Run  8 

0.16 

0.06 

0.04 

0.02 

0.03 

Run  9 

0.08 

0.08 

0.05 

0.05 

0.09 

Run  1  ) 

0.06 

0.11 

0.11 

0.10 

0.10 

Overall 

0.090 

0.071 

0.056 

0.052 

0.061 

We  see  that  20,  well  designed,  observations  enable  us  to  estimate  the 
true  percentage  response  curve  100  8{x)  to  within  under  an  estimated  6%,  of 
the  true  curve,  for  all  values  of  x  such  that  60%  <  100  0(x)  <  90%.  This 
estimated  accuracy  is  remarkably  row,  and  is  primarily  caused  by  the 
particular  estimation  technique  employed,  which  sensibly  compensates  for  large 
standard  deviations  of  the  maximum  likelihood  estimates  for  and  . 

Beyond  20,  well  designed,  observations  it  is  difficult  to  substantially 
improve  upon  estimation  accuracy  (together  with  good  design  efficiency) 
without  taking  a  large  number  of  further  observations.  For  example,  runs  3 
and  4  were  continued  for  a  further  30  observations  yielding 
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Trble  is  Estimation 

Accuracies 

for  up 

to  50  Further 

observations 

Sample  Si  ze 

25 

30 

35 

40 

'5 

50 

Run  3 

0.11 

0.06 

0.05 

0.08 

0.08 

0.09 

Run  4 

0.09 

0.12 

0.08 

0.08 

0.08 

0.09 

Fu.rthe  fiore,  for  all  ten  runs,  the  estimation  accuracies  do  not  noticeably 
increas..  fte.;  25  or  30  observations.  See  the  last  two  columns  of  Table  1. 

Our  t  1  tive  recommendations  for  the  practical  application  of  this 
procedure  are: 

(A)  Choose  up  to  ten  initial  obt  ’rvations  as  sensibly  as  possible,  by 
reference  to  the  practical  Vtuation  under  consideration. 

(B)  Obtain  20-25  fvrther,  ll-designed,  observations  based  on  the  Bayesian 
procedure. 

(C)  Then  stop,  unless  it  is  \n:  en  ed  to  proceed  to  a  sample  size  of  about 

100.  (in  this  case,  several  t.g.  5  new  desi  points  should  be  chosen  at 

each  time  stage.  These  should  be  oread  out  ac  ording  to  the  percentiles  of 
the  design  density.) 

The  percentage  design  efficiencies  fcr  the  te  .  runs  are  described  in 


Table  3. 


Run  1 

90.0 

93.3 

85.0 

96.0 

96.7 

Run  2 

40.0 

60.0 

70.0 

76.0 

80.0 

Run  3 

60.0 

73.3 

80.0 

84.0 

86.7 

Run  4 

40.0 

53.3 

65.0 

72.0 

76.3 

Run  5 

70.0 

80.0 

85.0 

88.0 

90.0 

Run  6 

100.0 

100.0 

100. 0 

100.0 

100.0 

Run  7 

60.0 

66.7 

75.0 

80.0 

83.3 

Run  8 

40.0 

60.0 

70.0 

76.0 

80.0 

Run  9 

90.0 

86.7 

90.0 

92.0 

93.3 

Run  10 

90.0 

93.3 

90.0 

92.0 

93.3 

Overall 

68.0% 

76.7% 

82.0% 

85.6% 

91.3% 

We  see  that  after  20  well-designed  observations,  82.0%  of  the  design 

t 

points  correspond  to  0.6  <  0(x)  <  0.9  for  the  true  curve.  This  excellent 
efficiency  increases  to  91.3%  after  30  observations. 

Therefore,  as  well  as  getting  good  estimation  accuracy  for  small  sample 
sizes,  the  results  are  based  upon  design  points  lying  in  the  important  region 
of  the  response  curve.  This  promises  excellent  robustness  under  changes  in 
the  sampling  model,  in  particular  when  these  changes  are  outside  the  region  of 
interest. 


< 
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3.  SEQUENTIAL  BEHAVIOUR  OF  DESIGN  POINTS 


f 


i 


Consider  the  sequential  behaviour  of  the  design  points  for  Run  1.  Here 
the  ten  badly  chosen  design  points,  discussed  at  the  beginning  of  Section  2, 
yielded  the,  rather  uninformative,  responses,  0,0, 0, 0,0, 0, 1 , 0,0,0. 

The  next  30  design  points  are  described  in  Table  4. 


Table 

4:  Desiqn  Points  for  Run 

_1_ 

igned 

Design 

True  Response  Curve 

Simulated 

ervation 

point 

0(x) 

y 

1 

0.84 

0.95 

1 

2 

0.70 

0.85 

1 

3 

0.64 

0.77 

1 

4 

0.60 

0.71 

0 

5 

0.70 

0.85 

1 

6 

0.66 

0.80 

1 

7 

0.64 

0.77 

0 

8 

0.70 

0.85 

1 

9 

0.68 

0,83 

1 

10 

0.68 

0.83 

1 

11 

0.66 

0.80 

1 

12 

0.64 

0.77 

1 

13 

0.62 

0.74 

1 

14 

0.62 

0.74 

1 

15 

0.62 

0.74 

1 

16 

0.60 

0.71 

1 

17 

0.60 

0.71 

0 

18 

0.62 

0.74 

1 

19 

0.60 

0.71 

4 

1 

20 

0.60 

0.71 

1 

21 

0.60 

0.71 

0 

22 

0.62 

0.74 

0 

23 

0.64 

0.77 

1 

24 

0.62 

0.74 

0 

25 

0.64 

0.77 

1 

26 

0.64 

0.77 

1 

27 

0.62 

0.77 

1 

28 

0.62 

0.77 

1 

29 

0.62 

0.77 

1 

30 

0.62 

0.77 

1 
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The  maixi mam  of  the  badly  chosen  design  points  is  0.39,  but  the  Bayesian 
procedure  moves  the  next  point  as  far  away  aw  0.84.  Then  observed  "ones"  tend 
to  push  the  next  design  point  downwards,  whilst  observed  "aeroo"  tend  to  push 
the  design  upwards.  2eros  create  mors  movement  than  ones*  since  these  are 
required  to  occur  between  proportions  of  0.1  and  0.4  of  the  time,  so  that 
their  less  likely  occurrence  has  greater  effect.  The  corresponding  values  of 
the  true  response  curve  move  very  quickly  to  the  region  (0.6,  0.9)  and  then 
stabilize  near  the  middle  of  this  region^  The  d-  /‘wards  movement  of  the 
design  point,  after  a  positive  observation,  »xvas  the  <r*8ign  into  a  region 
where  the  experiment  is  likely  to  be  more  inf c  r>.  .ive.  The  general  behaviour 
of  the  design  points  seems  to  be  highly  sensible. 

4.  THE  POSTERIOR  WFlviCT 

Note  that  after  the  30  designed  trials,  »,\n  1  fitted  maximum  likelihood 
estimates  of  -5.58  and  10.90,  for  8^  and  with  >'4 'proximate  standard 

deviations  of  2.15  and  4.17.  This  result  it  tyyicss'.l  of  all  ten  runs,  and 
suggests  large  discrepancies  from  the  true  values  tQ  »  -4.39  and  8.79. 
Nevertheless,  the  posterior  mean  of  the  response  curve  tas  in  close  agreement 
(estimation  accuracy  =  0.02)  with  the  true  cv  v%.  since  this  adjusts  the 
maximum  likelihood  estimate  of  the  response  curve  to  alluw  for  the  uncertainty 
represented  by  the  standard  deviations. 

This  dramatic  improvement  when  compared  with  cl ^.ssical  procedures  should 
perhaps  be  termed  "the  posterior  mean  effect." 


5.  COMPUTER  SOFTWARE 

Program  BIO&AIft . FOR  on  the  VAX  system  at  MKC  applies  computer  simulations 
(subroutine  PGMGEN.FOR)  to  the  design  and  estimation  procedures  (subroutine 


PGMBlO.FOR)  when  the  design  region  is  between  60%  and  90%.  Program  BlQ.FOR 
completes  the  design  and  estimation  procedures  when  either  (i)  the  design 
region  is  unrestricted,  or  (ii)  the  design  region  comprises  any  single  point 
lying  between  1%  and  99%. 
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